Linearized two-layers neural networks in high dimension

We consider the problem of learning an unknown function f⋆ on d-dimensional sphere with respect to square loss, given i.i.d. samples {(yi,xi)}i≤n where xi is a feature vector uniformly distributed and yi=f⋆(xi)+εi. study two popular classes models that can be regarded as linearizations two-layers neural networks around random initialization: features model Rahimi–Recht (RF); tangent Jacot–Gabriel–Hongler (NT). Both these also randomized approximations kernel ridge regression (with different kernels), enjoy universal approximation properties when number neurons N diverges, for fixed dimension d. specific regimes: infinite-sample finite-width regime, in which n=∞ while d are large but finite, infinite-width finite-sample regime N=∞ n finite. In first we prove if dℓ+δ≤N≤dℓ+1−δ small δ>0, then RF effectively fits degree-ℓ polynomial raw features, NT degree-(ℓ+1) polynomial. second both reduce methods rotationally invariant kernels. that, sample size satisfies dℓ+δ≤n≤dℓ+1−δ, fit at most features. This lower bound achieved by regression, near-optimal prediction error vanishing regularization.